# Description of Changes

Preparation for freeing empty pages.
Because the snapshot format depends on the density of page vectors and
didn't previously reserve a sentinel,
to preserve rollback safety we have to do this preparation step before
actually implementing freeing pages as part of the row delete operation.

In this PR, the `Table`/`Pages` switches to a `Vec<Option<Box<Page>>>`,
with pages allowed to be absent.
However, until a later patch, outside of tests, no page entry will ever
be `None`. The table code is still able to use and reason about `None`
page entries, as they may arise if we
deploy said later patch, free a page, capture a snapshot, then roll back
to this version.

In the snapshot format, absent pages are recorded in the pages vec as
the all-zeroes hash. Page objects are not written or read in this case;
the all-zeroes hash does not correspond to an actual object on disk.

When allocating a new page, we attempt to fill the lowest empty slot. We
do this in log time by storing a `BTreeSet` of the empty slots, and
popping the lowest value from it to use as the slot for the newly
allocated page. I believe that for at least some access patterns, this
should allow us to gradually converge on a dense array of pages in the
case where rows are deleted at a higher rate than new inserts.

As a result of this PR, some operators have changed in such a way as to
cause the `table` crate benchmarks to no longer compile. These
benchmarks appear to have already bitrotted significantly. I removed
benchmarks sufficient to get `cargo check --tests --benches` passing,
but did not attempt to repair `cargo test --benches`, as that appears to
have been broken prior to this change.

Per Joshua's review, this PR also adds a new metric,
`spacetime_replay_snapshot_num_absent_pages`.
Pages which aren't read from files due to having the all-zeroes hash are
counted towards that metric and not towards the existing
`spacetime_replay_snapshot_num_objects_read`.

# API and ABI breaking changes

Changes the snapshot format on-disk to recognize a new special sentinel,
the all-zeroes hash. When the all-zeroes hash appears in a table's
vector of page hashes, it means that no page exists in that slot, and so
no object file is read.

As of this PR, it is (or at least, should be) impossible to reach the
added codepaths or representation without using a snapshot created by a
newer version of SpacetimeDB or through manual editing.

# Expected complexity level and risk

3: if it is mistakenly possible to put `None` in a `Pages` within this
PR, then the release that introduces this commit may not be
rollback-safe. Also, this touches the datastore, which contains `unsafe`
code, though no new unsafe code is introduced, nor is any of it modified
in any ways that would affect safety invariants.

# Testing

<!-- Describe any testing you've done, and any testing you'd like your
reviewers to do,
so that you're confident that all the changes work as expected! -->

- [x] A couple new automated tests in the table crate.
- [x] Manually tested rollback safety by using clockworklabs#5771 to construct a
snapshot with an absent page, then used this PR to replay it. Described
in more detail in comment below.

---------

Co-authored-by: Zeke Foppa <196249+bfops@users.noreply.github.com>